The introduction of ICD-10-CM will increase the complexity of the diagnosis coding process, raising serious questions about comparability of data quality before and after October 2015. Specifically, extensive refinement of ICD-10-CM in comparison to ICD-9-CM and more complex rules give coders more valid alternatives in many situations, which may compromise the reliability of diagnosis coding with ICD-10-CM. Using a unique data set of de-identified text notes extracted from the electronic health information system at a large academic health system, and building on prior work converting quality measure specifications and estimating comparability ratios between code sets, we will (1) estimate and compare the inter-rater and intra-rater reliability of ICD-9-CM and ICD-10-CM diagnoses based on the same clinical information on patients who received intensive care, and 2) use the collected dual-coded data to investigate the content validity of ICD coding and the presence of missing pathways in General Equivalence Mappings (GEMs). The findings from this study will provide guidance to researchers and healthcare and public stakeholders regarding 1) possible challenges in longitudinal comparability of ICD-based quality indicators, clinical outcome measures, and surveillance measures, 2) changes that can be made to overcome data quality and comparability problems introduced by switching to ICD-10- CM, and 3) possible improvements in GEMs to benefit all data users.